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identified as having behavioral and/or academic problems. Participants 
completed a 27-day program with activities designed to provide opportunities 
to improve reading and math skills, develop positive social skills, share 
recreational activities, and share art and cultural experiences. Teachers 
participated by monitoring attendance and providing instruction. Analysis 
involved data on student demographics, academic achievement, attendance, and 
behavior. Results indicated that students' social skills development, report 
card grades, standardized test scores, and attendance were all positively 
impacted by attendance in the program. (Contains 30 references.) (SM) 
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Abstract 

In the spring of 1999, the Palm Beach County, Florida, Public School District implemented 
an after-school program with monies received from a 21st Century Community Learning Centers 
Grant. The goal of the program was to improve behavior, school attendance, and academic 
achievement. This program targeted “at-risk” students in one of the county’s high-need 
elementary schools. This paper reports on the results of the evaluation of that program. The 
results indicate that social skills development, report grades, standardized test scores, and 
attendance were all positively impacted by attendance in the after-school program. Teachers 
played an important role in program success through instruction and monitoring. It is important 
for teachers and teacher educators to recognize the impact teachers have not only during 
traditional classroom hours but in programs that run after the school bell rings. 
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Advocacy for All: 

A 21st Century Community Learning Center for At-Risk Students 

Introduction 

In January of 1999 the Palm Beach County, Florida, Public School District was awarded 
a three year 21st Century Community Learning Centers Grant from the U.S. Department of 
Education. A major goal of this grant was to improve behavior, school attendance, and academic 
achievement. Based on the findings of a comprehensive, local needs assessment conducted for 
the school system by the Palm Beach County Health and Human Services Planning Association 
(1997), three schools were identified as at-risk schools. This classification was a result of the 
presence of three indicators: (1) low academic achievement, (2) violent incidents, and (3) 
alcohol, tobacco, and other drug use. One of these three at-risk schools was designated to 
participate in a pilot program. 

In the spring of 1999, a comprehensive after-school program was implemented at the 
pilot school. The 63 students targeted for participation in the after-school program were 
identified by school administrators, guidance counselors, and faculty as exhibiting low academic 
performance and/or aggressive behavior. 

This paper reports on the results of the evaluation of the pilot program. In particular, it 
focuses on the results of one of the program’s goals: to improve behavior, school attendance, and 
academic achievement. The results indicate a need for teachers and teacher educators to advocate 
for programs for at-risk students. 

Literature Review 

A review of the literature was conducted to determine the benefits of after-school programs, 




to identify the characteristics of effective after-school programs, and to articulate the role that 
teachers play in successful after-school programs. 
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Benefits of After-Care 

Research indicates that almost all Americans feel that it is of importance to provide children 
with after-school programs that create a nurturing environment where they are able to develop 
both academic and social skills (YMCA, 1998). Children who participate in high-quality after- 
school programs increase their math and reading grades (Brickman, 1996; Chicago Public 
Schools, 1998; Gregory, 1996; Louisiana Department of Education, 1996; Posner & Vandell 
1994). In addition, they show higher levels of interest and ability in reading and math since 
literacy development increases as a result of practice and experience in after-care programs 
(Clark, 1989). Children in quality after-school programs also exhibit expanded development of 
new skills and interests (Gregory, 1996). Participation in after-school programs improves school 
attendance and homework frequency and quality (Carlisi, 1996; McCormick & Tushnet, 1996; 
Posner & Vandell, 1994; Riley, 1994). The literature also indicates that children in high quality 
after-school programs improve their behavior and are better able to handle conflicts. (Carlisi, 
1996; Gregory, 1996; Steinberg, Riley & Todd, 1993). Social skills development including 
improved interpersonal skills and the ability to cooperate with adults and peers is another benefit 
of quality after-school programs (Belle & Burr, 1997; Carlisi, 1996;Gregory, 1996; Terao, 1997; 
Steinberg, Riley & Todd, 1993). Finally, students’ self-confidence grows through development 
of caring relationships with adults and peers in the after-school setting (McCormick & Tushnet, 
1996; Steinberg, Riley & Todd, 1993). 

Characteristics of High Quality After-Care Programs 

The research indicates that high-quality after-school programs exhibit certain attributes. They 
are clear about their goals (Dept, of Education, 1998). They employ skilled and qualified staff 
(Vandell, 1995). They use community resources effectively including developing partnerships 
with law enforcement agencies (Fox & Newman, 1998). Additionally, effective programs 
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coordinate their curriculum with the regular school day and focus on providing a challenging 
curriculum in an enriching environment (Funkhouser, Fiester, O’Brien, & Weiner, 1995; Dept, 
of Education, 1998). Effective after-school programs meet many basic needs and support 
academic and social development by creating a safe and secure place for children after the bell 
rings. This is achieved by linking after-school activities with children’s classroom learning 
experiences. 

The Role of the Teacher 

The National Study of Before and After School Programs (Seppanen, 1993) found that 
human relationships were an important indicator of program quality. Quality relationships were 
defined as: warmth and respect between teachers and children, small groups to meet students’ 
needs, positive inter-student interactions, and team work among teachers and with parents. The 
notion of warmth and respect as part of quality relationships directly relates to teacher 
effectiveness. Characteristics of effective teachers have been defined similarly to those in 
Seppanen’s quality relationships and include: caring, understanding, being helpful/friendly, 
withitness, and social insight. These human relationship factors within the area of teacher 
effectiveness contribute to the creation of a climate that supports learning (Agne/ 1992; Gordon, 
1997; Wubbels, Levy, &Brekelmans, 1997). 

Effective teachers hold high expectations for all students, creating an environment that is bias 
free, instructionally effective, and thus promotes students’ achievement (Arends, 1998). Teachers 
with high expectations assign competence to students (Arends, 1998; Wlodkowski & Ginsberg, 
1995), thus helping them to develop a positive self-sense. Additionally, teacher expectations “are 
a vital part of motivating and effectively teaching currently low-achieving students” (Mac Iver, 
1992, p. 14). 
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Evaluation Design 

Theoretical Framework 

The theoretical framework used for this evaluation was an adaptation of the Tylerian 
Evaluation Approach, an objectives-oriented evaluation model. This approach was selected since 
the purpose s of the program were identified, and the focus of the evaluation was to determine if 
and to what extent these purposes have been met (Worthen, Sanders, & Fitzpatrick, 1997). 
Tyler’s model is composed of seven steps: (1) establishment of broad goals, (2) classification of 
goals (3) development of behavioral objectives, (4) identification of program activities, (5) 
selection of measurement techniques, (6) data collection, and (7) comparison of data results with 
objectives (Worthen, Sanders, & Fitzpatrick, 1997). This framework identifies discrepancies 
between performance and objectives. Since this was a pilot program, guided by pre-established 
goals, a modification of Tyler’s model provided the framework for evaluating program 
outcomes. This modification used the following steps: (1) articulation of the broad goal, (2) 
restatement of program indicators as behavioral objectives, (3) identification of program 
activities, (4) data collection, and (5) comparison of data results with objectives. By 
understanding how to use evaluation findings to improve student achievement, teachers and 
teacher educators can become advocates for all learners. 

Methodology 

Using the theoretical framework outlined above, the following section addresses the five 
evaluative areas. 

Articulation of Broad Goal 

The broad goal identified in this portion of the evaluation was stipulated in the 21st 
Century Community Learning Center’s grant as a focus for annual reporting (U.S. Department of 
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Education, 1998). Thus the goal of improving behavior, school attendance, and academic 
achievement guided the development of program activities and evaluative measurements. 

Restatement of Program Indicators as Behavioral Objectives 

Eight program indicators were defined in the 21st Century Community Learning Centers 
Program Annual Report (U.S. Department of Education, 1998). Two indicators were used to 
develop behavioral objectives. First, students regularly participating in the program show 
continuous improvement in achievement through measures such as test scores, grades, and/or 
teacher reports. Second, students participating in the program show improvements on measures 
such as school attendance and decreased disciplinary actions or other adverse behaviors. 

Translating these indicators into behavioral objectives allowed for data analysis. Thus the 
indicators stated as objectives became 

(1) Students will meet or exceed the district minimum standard on the reading portion of 
the CTBS. 

(2) Students will meet or exceed the district minimum standard on the math portion of the 
CTBS. 

(3) Students will increase their report card grade/grade level in Reading half a 
grade/grade level or more. 

(4) Students will increase their report card grade/grade level in Math by half a 
grade/grade level or more. 

(5) Total participation in the program will reflect a 90% program attendance rate. 

(6) Students will demonstrate a significant decrease (p<.05) on the internalizing portion 
of the Achenbach scale. 

(7) Students will demonstrate a significant decrease (p<.05) on the externalizing portion 




of the Achenbach scale. 
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(8) Students will show improved social competencies skills as demonstrated by 
significant increase (p<.05) in pre and post SSBS scores. 

(9) Students will show improvement in their antisocial behavior as demonstrated by 
significant decrease (p<.05) in pre and post SSBS scores. 

Identification of program activities 

Program activities were designed to support successful implementation of the objectives. 
Participants completed a 27-day program (81 hours) which met five days a week between 2 
P.M. and 5 P.M. each day. Program activities were designed to provide students with an 
opportunity to improve reading and math skills, develop positive interaction with others (social 
skills development), share recreational activities, and share art and cultural experiences. All of 
this was provided in a safe and drug-free after school environment. 

Under the direction of the Safe Schools Center, a project director and certified teachers 
were hired to supervise the daily operations of the program. Activities designed to improve test 
scores and report card grades involved teachers providing math and reading assistance using 
materials prepared by the Safe School Center as well as materials purchased with grant money. 
Alternative teaching methodologies were encouraged. Guidance counselors and community 
agencies (e.g., Department of Parks and Recreation, Out of School Consortium) provided support 
and assistance with recreational and cultural activities. 

Attendance was monitored by the teachers who kept a log of student attendance and 
followed this with phone calls to the parents of students who had missed a daily session. A social 
skills curriculum was designed to address student behavior. This curriculum included skill 
development in resiliency building, coping, anger management, and conflict resolution. 
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Data collection 

Quantitative data were collected on student demographics, academic achievement, 
(standardized test scores, report card grades), attendance, (program attendance logs), and student 
behavior (The Achenbach Teacher’s Report Form, The School Social Behavior Scales). 
Additionally, demographic data were collected on participants (district reports). 

Instruments 

Participants’ Demographics and Economics. The demographic and economic data were 
collected from the district’s data bases by the program coordinator. Participants were identified 
by their student identification numbers to protect anonymity. Data were loaded into the SPSS 
computer program for purposes of analysis. 

The Comprehensive Test of Basic Skills (CTBS). The CTBS is a standardized, norm- 
referenced test annually administered by the school district. Since standardized test scores are 
often used to assess students’ academic performance for local, state, and federal level 
comparisons (Arends, 1998; Burden & Bryd, 1999), CTBS scores were selected as one measure 
of academic performance. Changes were calculated as the percentage of increase or decrease in 
the test scores from the 1997-1998 academic year to the 1998-1999 academic year scores. 

Report card grades. The second measure of participants’ performance was report card grades, 
since they are typically used as an indicator of students’ academic performance. For example, 
report card grades determine a student’s promotion to the next grade or retention in the same 
grade (Burden & Bryd, 1999). Since the pilot school ran a Montessori program, all students did 
not receive report cards. For those students who did receive report cards, grade differences 
between the first and the third marking period were reported. For those students who did not 
receive a report card grade, increases in grade level between the first and third marking period 



were reported. 
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Attendance logs. Participants’ attendance rates were calculated as the number of days 
attended over the number of available attendance days in the program. Measuring attendance in 
this way avoided penalizing participants who were late entries into the program. 

The Achenbach Teacher’s Report Form. The Achenbach Teacher’s Report Form is an 
instrument typically used for identification of students’ behavior problems (Achenbach, 1991). 
The student’s classroom teacher completes the form, which includes a student profile section for 
anecdotal comments and a behavioral rating section. The Achenbach scores for the behavioral 
rating section are reported as an internalizing total and externalizing total, thus identifying the 
student’s expression of antisocial behaviors as turned within toward his/herself (depression) or 
turned outwards toward others (aggressive behaviors). Participants were rated upon entry into 
and at the close of the program, with scores on the pre and post test measures tested for 
significance of change at the .05 level using the paired-samples t-Test. 

The School Social Behavior Scale (SSBS). The School Social Behavior Scales (SSBS) 
provides a valid and reliable, standardized measure of students’ school behavior (Merrel, 1993). 
The student’s classroom teacher completes the scale, rating the student on social competence and 
antisocial behavior. The social competence dimension assesses interpersonal skills, self- 
management skills, and academic performance yielding a social competence total. Antisocial 
behavior is assessed via behaviors: hostile-irritable, antisocial-aggressive, and demanding- 
disruptive yielding an antisocial behavior total. Participants were rated upon entry into and at the 
close of the program, with scores on the pre and post test measures tested for significance of 
change at the .05 level using the paired-samples t-Test. 

Comparison of data results with objectives 
The following section, findings and conclusions, presents a comparison of the data 
analysis with the program objectives. Discrepancies are noted, suggestions for further program 
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development are offered, and implications for teachers and teacher education programs are then 
presented. 

Findings and Conclusions 

Demographics 

There were 63 students enrolled in the pilot after-school program. Thirty-one (49%) 
students were in the fourth grade and 32 (51%) were in the fifth grade. There were 35 
male participants (57%) and 28 female participants (44%). The school district report 
reflected 52 black students (83%), 7 Hispanic students (11%), and 4 white students (6%). 
Most of the students (58, 92%) received free or reduced; 5 (8%) did not. 

Twenty-five students (40%) were reported as dropout prevention program participants. 
Nine students (14%) were Special Education Students (ESE) students and 4 (6%) were English 
for Speakers of Other Languages (ESOL) students. Twenty-five students (40%) were reported as 
not participating in any special programs. 

CTBS Scores 

The Comprehensive Test of Basic Skills (CTBS) math and reading scores were analyzed 
by comparing scores for the past two years. Data are reported in Tables 3 and 4 as an increase in 
scores, a decrease in scores, or no change in scores. 



Table 3 CTBS Reading Scores 





Number of Students 


Percent 


Increased Scores 


30 


59 


Decreased Scores 


16 


31 


No Change in Scores 


5 


10 


Total 


51 


100 


No Score Reported 


12 




Total in program 


63 
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Table 4 CTBS Math Scores 





Number of Students 


Percent 


Increased Scores 


33 


65 


Decreased Scores 


15 


29 


No Change in Scores 


3 


6 


Total 


51 


100 


No Score Reported 


12 




Total in program 


63 



The majority of students who participated in this program had increased CTBS reading 
scores (59%) and increased CTBS math scores (65%). Most of the participants in this program 
were targeted because they scored in the bottom two quartiles on the CTBS. These initial results 
seem to indicate that the students who participated in the after-school program benefited from the 
extra help in math and reading. Unfortunately, no district minimum standards were available for 
either the math or reading scores to measure success in terms of the stated behavioral objectives. 
Report Card Grades 

Academic improvement was also measured by improvement in report cards grades 
between the first and third 9-week marking period. During this time, 39 students (75%) increased 
their math grade one grade or grade level. Decreased math grades/grade levels were found for 3 
students (6%) while 10 students (19%) neither increased nor decreased their math grade/grade 
level. Similar findings were reported for Reading/English grades/grade levels. Increases were 
reported for 38 students (73%), while 4 (8%) students decreased their grade/grade level. Again, 

10 students (19%) neither increased nor decreased their Reading/English Scores. The immediate 
effects of the daily reinforcement in math and reading skills may have been a contributing factor 
of this improved academic performance. 
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Attendance 

The pilot program ran for 27 days. The average attendance rate for the 63 students in the 
program was 88%. Table 1 shows attendance rate (percent of days attended) by frequency. 



Table 1 Attendance 



Days Attended 


Number of Students 


Percent of Students in Program 


27 


14 


22 


26 


12 


19 


25- 


10 


16 


24 


8 


13 


23 


4 


6 


22 


1 


2 


21 


3 


5 


20 


3 


5 


19 


3 


5 


17 


1 


2 


15 


1 


2 


14 


1 


2 


13 


1 


2 


11 


1 


2 



While the 88% fell somewhat short of the 90% stated in the objective, the results of this 
pilot study were extremely encouraging. In addition to the high daily program attendance rate, 
less than 1% of the participants had excessive absences (i.e., over 10 days). 

Aschenbach Teacher’s Report Form 

The Achenbach Teacher Report Form is used for the identification of students’ behavior 
problems. Scores are reported as an internalized total and externalized total, which identifies the 
student’s expression of antisocial behaviors as turned toward self or turned outward toward 
others. Results of paired sample t-tests found no significant change in student internalizing and 
externalizing behavior at the completion of the program. Program duration may have played a 
part in the failure to see significant reduction in pilot participants internalizing or externalizing 
behaviors. 
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SSBS Results 

A second measure of program effectiveness utilized in this study was the SSBS, a valid 
and reliable instrument that uses two scales to evaluate students’ social competency skills and 
antisocial behaviors. Each scale is broken into sub-scales. 

Data analysis using a paired sample t-test found a positive statistical significance (p<.05) 
in the following areas: Total pre/post social competency skills (p = .001), Interpersonal skills 
(p = .001) and Self-management skills (p = .049). Table 2 shows both of the scales and the sub- 
scales, the mean scores for the pre- and post- test and p value. This data represents 44 out of the 
63 program participants who had both pre- and post-test scores. 

Table 2 SBSS Results 



SBSS Category 


Mean 


Sig. (2-tailed) 


Total Social Competency.- Pre 


104.55 


.0001* 


Total Social Competency.- Post 


114.07 


Interpersonal Skills - Pre 


42.32 


.0001* 


Interpersonal Skills - Post 


48.23 


Self-management Skills - Pre 


34.68 


.049* 


Self-management Skills - Post 


36.61 


Academic Skills - Pre 


27.50 


.070 


Academic Skills - Post 


31.50 


Total Anti-social Behavior - Pre 


64.89 


.906 


Total Anti-social Behavior - Post 


65.15 


Hostile-irritability - Pre 


29.77 


.627 


Hostile-irritability - Post 


30.39 


Anti-social aggressive - Pre 


17.89 


.815 


Anti-social aggressive - Post 


18.07 


Demanding-disruptive - Pre 


17.52 


.306 


Demanding-disruptive - Post 


16.73 



* p<.05 




It appears that the Social Competency Skills of these students have improved overall, 
while the Anti-Social Behavior has made no significant shift. These data are representative of a 6 
week pilot program. Within that limited time, frame it is encouraging to see the positive shift in 
Social Competency Skills. However, the lack of shift in Anti-Social Behavior should not be 

15 
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taken to indicate that the program is ineffective in this area. Research has found that it takes a 
much longer period of time to modify anti-social behaviors. As is evidenced by the decrease in 
the demanding-disruptive mean scores, there has been some progress in this area. 

CTBS scores, report card grades, attendance rates, and SSBS results, all show positive 
growth on the part of these students. While this is early data, it appears that the 21st Century 
Community Learning Centers Program has the potential to positively impact the academic and 
social lives of the students who it reaches. 

Implications for Teachers and Teacher Educators 

Implications for teachers and teacher educators can be drawn from the literature on 
teacher effectiveness. One of the true success stories of this program is the high attendance rate. 
Teachers were responsible for tracking attendance and making follow-up calls to the parents. 

This was an important element that contributed to the success of the pilot programs. The 
literature indicates that regular attendance is linked to academic success in remedial programs 
(Bender, Giovanis, & Mazzoni, 1994). Both standardized test scores and report card grades/grade 
level improvement reflected this. 

After-school teachers worked with the classroom teachers to link the program activities 
with classroom activities. They developed relationships with parents, students, administrators, 
and support staff to create a climate that supported learning (Agne, 1992;Gordon, 1997; 

Wubbels, Levy, & Brekelmans, 1997). As a new generation of teachers enters the nation’s 
classrooms, it is important to recognize the role they will play in children’s success both during 
the school day and after the bell rings. 
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